Learning in Restless Bandits Under Exogenous Global Markov Process

We consider an extension to the restless multi-armed bandit (RMAB) problem with unknown arm dynamics, where exogenous global Markov process governs rewards distribution of each arm. Under state, evolves according Markovian rule, which is non-identical among different arms. At time, a player chooses out $N$ arms play, and receives random reward from finite set states. The are restless, that is, their local state regardless player's actions. Motivated by recent studies on related RMAB settings, regret defined as loss respect knows dynamics problem, plays at time $t$ maximizes expected immediate value. objective develop arm-selection policy minimizes regret. To end, we Learning under Exogenous Process (LEMP) algorithm. analyze LEMP theoretically establish finite-sample bound show achieves logarithmic order time. further numerically present simulation results support theoretical findings demonstrate significantly outperforms alternative algorithms.